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ABSTRACT 

Knowledge of results (KOE) is most frequently cited 
as the reinfcrcer in self -instructional systems. The printed answer 
in a programmed text, for example, is supposed to reinforce the 
response the student emits previous tc observing that answer. Some 
other possible reinforcers are briefly discussed in this paper before 
the literature cn KOR in self-ins true ticn was selectively reviewed. 
The review was organized as a search tor evidence that KOR might 
appropriately be called a reinfcrcer. Studies comparing programs with 
and without feedback were examined; the weight of evidence from these 
global studies was that feedback did not enhance learning, as 
measured by immediate post-test scores cr by retention tests. In at 
least one case there seemed to be a decrement in performance 
traceable to the presentation ct feedback. Studies in which 
"schedules of reinio rce me nt " were varied similarly failed to show 
effects that would be expected it KCR were acting as a reinforcer. 

One major study involving delay of KOE did report the effect expected 
when delivery cf a reinforcer is delayed. Other studies on delay do 
net replicate this finding. Finer grained analyses of student 
behavior and KOR begin to reveal specific conditions under which KCR 
seems to re acting as a reinforcer. (Author/AMM) 
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KNOWLEDGE OF RESULTS AND OTHER POSSIBLE REINFORCERS 
IN SELF- '.’.'STRUCT IONAL SYSTEMS 1 

*} 

George L, Geis and Reuben Chapman 4 " 

Center for Research on Language and Language Behavior 
The University of Michigan 

The knowledge of results literature was reviewed. It 
was concluded that printed answers in programmed texts do 
not automatically or globally function as reinforcers. 

Conditions in which information on one's learning perform- 
ance becomes reinforcing were identified for future research. 

An important contributor to the technology of education has been 
reinforcement theory, or popularly, "Skinnerian Psychology. 11 Therefore, 
discovering, making explicit, and developing reinforcers in learning 
situations might well be a major are^ of concern to the technologist, 

Though this paper concentrates, in a sense, on the search for rein- 
forcers it is recognized that unsystematic contingency setting can, even 
with the most powerful reinforcers in hand, result in poor learning, 

B. F, Skinner (1968) points out that "...it is not the reinforcers which 
count, so much as theii relation to behavior. In improving teaching it 
is less important to find new reinforcers than to design better contin- 
gencies using those already available," Nevertheless the explication of 
available reinforcers seems a necessary step in developing more effective 
contingency management in learning. 

Special attention should be paid to reinforcement by those engaged 
in examining and designing self-instructional systems. Traditional 
reinforcers ; n education aro often intimately tied up with the teacher/ 
classroom system. The teacher is more than an exposer of material; 
sometimes he mediates teinforcers (e.g., reports activities to the 
principal, seats dyads end triads cf students in certain places). At 
other times he directly administers the re* inforcers (e.g., praises one 
student and applies the ruler to the hand of another). The designer 
who eliminates the teacher from his system is rt reving a reinforcing 
aren: who has at leas: the potential for setting up subtle cont ingencies 
and for varying reinf crcers with the moment ar.d the student. 

Tnis paper briefly discusses seme potential reinforcers for self- 
instructional systems and then provides a detailed review cf the 
literature relevant to one cf them: knowledge of results. 

Sore Possible heir, forcers 

llaser f 1965) suggests an empirical approach for determining rein- 
forcers: ,f Re inf rreing events rust be determined on the basis of detailed 

analysis cf appropriate subject matter and component repertoire relation- 
ships [p, per V but be proposes seme "illustrative leads'’, starting 
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with an application of the Premack principle (Prenack, 1959) which 
suggests that high robability behaviors can be used to reinforce 
behaviors which have a lower probability of occurrence. Glaser cites 
one example. In a .!■ arning situation "backward chaining' 1 would allow 
the student to se reinforced wit hi the high probability, i.o., first 
and better learned behavior when he emits a previous step in the chain. 
Thus, teaching the student Step C all, c- ■ s it to he a reinforcing event 
for emissions of Step B when it is being learned. A variety of ether 
educational applications of the principle have been demonstrated (see, 
for example, Osborne, 1969). 

Man ipulat lo r, or demonstrations of master y or compe t e nee have often 
been cited as reinforcers or, in other terminology, as strong motives 
(see, for example, Waite, 1959). Glaser suggests "overt control over 
the environment" as a second possible reinforcing event. Skinner (1968) 
has suggested a similar class oi consequences be considered reinforcing. 
"Children play for hours with mechanical toys, paints, scissors and 
paper, noise-makers, puzzles — in short, with almost anything which feeds 
back significant changes in the environment and is reasonably free of 
aversive properties. The sheer control of nature is itseif reinforcing 
[p. 20]." 

Many self-sustaining tasks, such as science learning kits or educa- 
tional construction toys, seem to have this feature, Self- inst rue tional 
materials might provide the chance to manipulate materials either during 
learning or as a consequence of successfully completing a learning assign- 
ment, Moore and Andersen (1968), Gctkin (1966) and others have linned 
heavily upon the reinforcing effect of manipulation to sustain the 
learner's behavior in their "responsive emv i ronment s ■ 11 

Discovery , cur ios i ty a r.d e xp lorn t ion v : \ i c k h ,_i v e oft e r h o e n p elated 
to as reinforcers or motives seer to be similar to the event 5 just 
discussed. For instance, one of the supposed benefits of tue "discovery 
learning method" * s the sustained interest and r ot i. va ti on o. r... ted by 
the activity of discovering. (In all ease- being cited h re, the specific 
and detailed evidence is lacking to support the contention that a certain 
class of events _is_ reinforcing, he attt-.pt is leir.g m- .e here- to validate 
claims; this is merely a catalog of proposed and potential re in ; ere ers 
for self-instructional systems.) 

Self- instruction may te narrowly interpreted to mean "teacher less . M 
In that case social reinforcers ether than those mediated :v a teacher 
night be considered. For ex —pie, : :,l Far an Tt-vo 1 c r r.ent Institute (1966) 
h a ? p rod need d y ad i c p r c grams ( i n vc 1 vine t w ^ 1 e a r ne r s '■ . A ; 1 1 f - i r. s t r u c t 1 c n 
orcbl*m driving course frr 1 e n c : * t ^ (Gei>, et at.. 1969) includes 
exercises which utilise two t lt.l h*. rs ir.t - 1 rv i ew; c each other, , edited ty 
checklists in the text. The "jrhos" which are an 1-pcrta; t part :f a 
recent text beck fferster l V err c t : , 19fM require tv; students cot p era ting 
on the exercises. In such cases us tuest it is certainly possible that the 
generalized rcinforcer* provided by an. t her j true may sustain stucent 
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participation in the learning tasks. Furthermore specific reinforcers 
supplied by the partner (e.g., "Now you're doing it correctly") may 
more precisely control learning. (In descriptions of his partially 
self-instructional system, Keller has stressed the reinforcing effects 
of peers who act as student "proctors." See Keller, 1967, pp. 17-19.) 

Extrinsic reinforcers ranging from the omnipotent M and M candy to 
points, tokens, and toys, have been used as reinforcers in many behavior 
modification studies and recently have been used in connection with self- 
instructional materials (see Berman, 1967; Smith, et al., 1969; Sullivan, 
et al., 1967). Systems for reinforcer delivery, more elaborate than 
those generally present in a text or kit, are usually required when such 
reirforcers are used. Either a human banker or some thief proof equipment 
may be called for. 

Progress itself has been proposed as reinforcing: evidence of moving 
toward a goal may be sufficient to sustain the learner. Progress is 
defined a bit more specifically by Caber, et al . (1965). "Knowledge 
alone of the fact that he is performing correctly may be a less 
effective reinforcer for the student than being permitted to engage in 
further and mere complex activity .... Being able to move on, to get into 
and discover the fine details of the subject matter without being incorrect, 
frustrated, or punished for being wrong may be the most potent reinforcing 
consequence in a programmed instructional sequence [p. 10]." Record 
keeping and progress plotting is common among behavior modifiers and has 
been incorporated into a number of self-instructional systems. 



Aversive stimulation has not been extensively explored either in the 
basic research literature of psychology r.or the technology of education. 
v et the continuing evidence from everyday living suggests it is a frequent 
and powerful controller of human behavior. Some branching programs 
incorporate verbal punishers in the text. Students who choose incorrectly 
on & multiple choice frame r.a^ be directed to a page which contains a 
verbal rebuke. It has been pointed out elsewhere that "...finding a short 
cut or an easier way to emit a certain response is also reinforcing for 
most learners [Taber, Glaser 6 Schaefer, 1965, p. 27j." Some experimental 
evidence supporting this modern restatement of the Law of Least Effort is 
available. The results of one small study (Gets 6 Knapp, 1963) indicate 
that avoidance of additional work in a program, i.e., reduction of the 
number of frames the student is required to cic, can serve as a reinforcer. 



Possibly aversive stimulation is involved in the concept of reduc t ion 
of tension or of uncertainty. Many people (see Berlyne, 1960) have 
suggested this as a major variable in the control of human behavior, 
especially human learning. A learning situation, it is contended, should 
produce a slight rise in tension followed by tension reduction after the 



(Seemingly related is the concept of c losure or 

by Gestalt psychologists.) The authors of a 3argc 

reading program for children (Smith, I 9*4) have informally commented on 
the importance c; this vaveiike tension profile in self-instruction. 



response is emitted 

c nmp i e 1 1 on a c v a n c c c 
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Presumably they have tried to design tuoir programmed materials accordingly. 
Less precisely controlling sLud-or.t uncertainty but prosably based upon the 
same assumption are those programs which the authors claim have error 
production purposely built into their in order to "maintain student interest." 

The most frequently cited reinforccr i:i the Literature on self- 
instructional systems , especially programmed instruction, is Knowledge 
of results (KOR) . Almost all published programs, as veil as innumerable 
articles and texts cite as one of the rules of programming: "Include the 
correct answer in the program in order tc- reinforce the learner." 

The remainder of this paper will be devoted to a discussion and review 
of the literature cn this, the r.o«t popular, candiuate fer the role of 
reinforcer in self-instruction. 

The phrase "knowledge of results" at first glar.ee seems self-explanatory 
and specific. However, it refers to a great variety of environmental changes 
or stimulus presentations ranging, in. instruction, from incicatir.g to tr.e 
learner that he is correct to providing elaborate, informative, corrective 
materials when he has made a mistake. "Non-verbal" leeccack cor.sec uerices 
are as varied. A stuoent might discover hew well he has assembled a piece 
of apparatus by plugging it iu and trying it out; he might test a formula 
he has invented by going into the lab and bii-ad-ng the ingredients; he 
might observe a computer sirul at L mi*, of hi? patient's vital signs before, 
curing, and after the treatment he ins propose-.; j h.e might watch u model ot 
the bridge he has designed u I sp.l a;, cci cr. a cutnuju r.iy tube ant undergoing 
stress from traffic and wines. Lburc feedback c. n sequences are non-verbal 
but do not occur "naturally" as a c usequer.-.. e of the student's previous 
performance . These have already been re; ticucJ under the near, in.g ot 
extrinsic re inforcer s . 



Ir. fact, any consequf ncc d isc r i mi nnbi e Co the learner ur.u regularly 
related to his previous performance Cc.n e des i gnat i d KOP . A child 
working on a teaching machine viuin preduv. s a >:reo n 1 yght uui : tire no 
makes a correct response, is netting ftednauk. Ti.t child who :s told tr.at 
he may go to the playground when he tini-hc^ his assignment is also in a 
feedback situation. Sc is th-.» student vhc , after ne M*- answer .in 

class bears the teacher say, 'Well, yet. hut does .-.nycre e^se :.ave a 

di f f e rent answer to suggest?" And so is t:.e y..iid vno hears increasing 
giggling from his peers as he works the p rubier, on the bl a rkbuaru . 

But KOR usual iv r. e an s s c r e t h n «•. r ; re specific t r . a n t h a t . more p c r - 

fonr.ar.ee consequences ere return-ruing iu and of t ht - re 1 Ye - . Mipp use* 
that a child is pornititd to play with a t-. y vh^n he has rlrishtd spelling 
five words correctly. Tleyine with tie toy is a r't t u : c rc m,g Uwttvtty. 

It can be used to strengthen. j "o r ; t y L -i b iviirs; it is ; . : t unique. y 
tied to the task at hand. Conr^'t it with a ;r , iu wr.tcn, utter 

spelling each word :ht -tuuir.t is sk.vu tr.e tf. :i it spelling - ‘..at wire. 
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Permission to play serves the same inf o rrr.at ive function as the correctly 
spelleu words but it is also reinforcing "in its own right." 

A further distinction has to do with the specificity of contingencies. 
Usually K.0R refers to consequences immediately after each response. It is 
often contingent upon approximations tc a tr.ore complex final performance. 
Other "reinforce rs" are usually contingent upon completion o: a 1 .rger 
task (e.g.. an assignment rather than a frame in a program). 

KOR is often a mediating system; a means to an end. The child who 
is promised permission to play when he has finished the assignment may 
also have been told he was correct (K^R) after each word or problem. 

It is sometimes said that being correct or "knowing you are right" 
is reinforcing in itself and need not be linked tc other reinforcers. 

This claim for ar. autonomous status of KOR is particularly common among 
educators who are concerned that the use of other reinforcers either 
instead of or in addition to KOR not only weakens the effect of "the joy 
of learning for its own sake" but also smacks of immorality and bribery, 

Ve will not pursue questions concerning what particular type of re inforcer 
KOF may be but will confine ourselves to discovering any evidence that it 
is a reinforcer at all. 

We will limit the use of "KOR" tc those cases in which the major 
consequence of the response is information about the learner r s own per- 
formance. Ve are excluding all of the potential reinforcers dealt with 
earlier in the paper which have demonstrable reinforcing effects outside 
of the task situation. 

So far we have run the risr 
variety of terms interchangeably 
a n d k n o w 1 e ci g e o f r e s u 1 1 s . 

bach of these terms has its drawbacks. F e c d b a c k , for example, 

suggests continuous guidance and an adjustment to the responses producing 
the feedback; the speedometer c n an automobile illustrates feedback 
better than a bit of programmed text. Michael and Maccoby (1961) have 
noted that "the term; ’feedback* aid other related expressions ... are , in 
a sense, misnomers, as applied to the procedures of the kind used here 
(providing answers in a programmed situation) in that the information 
is the s am.-: , f r cm, a s : i mu 1 us p c i r. t c f view, regardless of what response 
the subject has made during practice. In. the knovledge-cf-result s 
procedure, as traditionally cppliej to practice cf skills, by contrast, 
...feedback depends u; on the extent anc; kinds of errors just made in 
practice, In both ins t ar.c ls , how., cm r , . . . ’ f et :iback ’ does provide a basis 
eitner for r e c n i ? i n e and correcting trrors made in practice [footnote, 
p, 162'." In the context cf Ft 1 f- i r.s t rue t ional systems, the word "feet- 
back" r. ieht mere at; ropria U ly ‘r c used t: refer tc response consequences 
in computer-based branching programs. 



t of appearing ar.ti-seir.ant ic by using a 
: feedback, confirmation, reinforcement 
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"Cor i l mat ion 11 , 0:1 tif other hand , s u o g< 1 -it: s that a correct response 

preceded answer-observing. Th eu ,;h an error cars, in a sense, be confirmed , 

general usage ind i rates a positive ro 1 a t i uns : , 1 p between response and con- 
firmation, making the word synonymous, with "verify" and "validate." 

"Reinforcement 11 presupposes r h ; ■ re 1 a i enr i : i p between answer and 
response that is being qi>' d irnci in ih i - purer: the purpose of this 

paper is to explore the pos =■. 1 b i 1 •' r v t! it the p re sen ta t i on of an answer 

contingent upon, a .student le-p-.atv a ra-in i'erscv, 

"KOR" has a slight mental Is t i c tinge ( L . e , , <ioo-> the word "knowledge" 
indicate some mental activity of the >> laden t ? 1 . Neve r t holes s Lt seems 
like the least ami: iguous term and ■::!] genera! iv h** fed throughout. 

Behind the p roros i t i or. that KUK (fn., the or in tod answer in a progrcn,) 
is a reinforcer lie many no-sumpt i •vis a'-n the fudeot's in to rae t i c r with 

that answer, 

Suppose that a student i -> vo oh L:v,: his wav C i i v v . y h a programmed text 
which supplies the noire.* t anwr in printed form f at to each item. 

“Gnat printed answer cannot h-’'v*» on •. f*h ct on t : : * ■ siuient unless ; e cm' es 
in cent act wif it, (fn. cra:,":*w •; h o be»-*i are i alternately of being 

too artistic and loo scientific 1 > u i no -vim r , , charged them with 

being me t apdiy s [ -va 1 , *' nbsvrwti r. *; -f stu*'*: t.- ■ r-' nr, through programmed 

materials, to he d :sn.n sed at Uufth 1 iter, Micaui fat the learner does 
not always Ipuk at t' printed e.fW"r. • r -v ; . ' i a g printed answers then, 
is not reinforcing, i r. .ipd-'w f r an ar.-v ■ r m. • cc me so, the student 
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student who, by observing answers in a program, repeatedly discovers that 
his own answers are erroneous and who nevertheless continues to observe 



This short discussion indicators that the easy phrase "the answers 
are reinforcing 1 ' ought to be approached with caution and accepted only 
tentatively until a better definition and mere evidence are provided. 
Such conservatism will be reinforced by an examination of the relevant 
research literature, 



Most studies (including those reviewed here) are not directly aimed 
at investigating whether or not answers are reinforcers, The question 
usually being attached is a broader one: Dees feedback in some way affect 
performance during and after self- (programmed) instruction? 

Feedback may affect performance, but technically not be a re inforcer, 
serving some ^ther function in learning. Goldbeck and Briggs (1962), for 
example, suggest that feedback "...nay provide information concerning the 
adequacy of responses mace, may serve, as a reinforcement and reward for 
responses, may have a motivating effect or performance, may be used to 
direct the next step to be taken in the learning program [p, 184 j." 

Just as it might be mistakenly identified as a reinforce*, when it 
was not, KGR could act as a rein forcer for behaviors not being measured 
by the experimenter and so be overlooker. For example, locking at an 
answer in a programmed text may be reinforced by "seeing it." But if 
such answer-observing does not lead to better p^st-test performance (and 
post-test performance is being measured in the study), jne might 
erroneously conclude that KOR does not function as a reinforcev at all, 

in this paper feedback studies will be reviewed 'with an eye to the 
more specific problem: Is KOR a reinforcer in self-instruction? The 
review is restricted almost entirely to self-instructional situations. 

It has already been pointed cut that there is a large literature on KOR 
which represents res card u^ing other than programmed instructional 
materials. (For a tender survey, the reader should consult a recent 
comprehensive berk (fvr.net t, 19*9) which .deals with a great variety of 
tasks involving f e e c Ki Tx . ) VMle such research seems occasionally to 
threw light on KOR a.- a variable in learning, often the situation differs 
sc much from s e 1 f-ir. - t ru: t i c n that extrapolation is unjustified. 

Feedback vs, N c- r e e d h a c k 

(As Ann e c t '196 r O prints cut, intrin s ic feedback :$ unavoidable; 
it recurs as part c f t n t perf" rn a r, z e itself, e , e • , nearing y c u r own 
evert vocalizing, "S'c- feedback" rer.r.« nr exp eri -ent er-suppl ■ e d feedback.) 

there is a~;le evidence that under -cr.e circumstances feedback af f tct 
p £ r i r rmar, ;e , however, the re-ults with regerd to prcgra~med instructicn a 
S ; m £ v. ; , a t amt I E J C u > . 



the printed answer rriight be suspected of masochism.) 
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An oft-cited study by Feldhusen and Birt (1962) used a short program 
presented to college students. The authors concluded that the. no-ccnf irma- 
tion groups did not significantly differ from the confirmation group, 

Moore and Smith (I960 also reported no differences on the post. -test 
between KOR a..d r.o-KOR groups of college students who used a version of 
the Holland-Skinne r psychology program. The experiment era tried a 
variety of feedback conditions (KOR alone, KOR plus pennies, KOR plus 
light), none of which seemed to affect learning significantly, However, 
errors within the program were fever for the KOR groups. 

In an earlier study by the same authors (Moore & Smith, 1961) a 
spelling program for 5th- and 6 th-grade children was presented with and 
without confirmation* Again, there seemed to be no difference in terms 
or achievement test results between the two treatment groups. 

Hough and Revs in (1963) also reported no effect of KOR when it was 
used in a 555-frame college ievei program* Echoing a recurrent theme 
found in this literature, the-.v state: ,f Tns program used in this study 
has a lew error rate. When students ’know' that their resp rise is 
right, and thus presumably reinforce themselves, it would seer, reasonable 
that further confirmation in the form of a reinforcement frame would be 
redundant fp. 290]." 

Bee k r (’1 9 ^ 4 ) us c d a 16 
small croup of adult learners 
dramatic ga ' n scores, the Hit 
sign’ficart. 

While finding? wh.ich indicate no effect cf feedback are disconcerting, 
even more damagin,- would he evide:c_ that feedback actually hindered per- 
formance, ?wtt c , e t al, (1962') report such data. were taucht to 

identify r.ul tied men? ; onal , n n-verbnl sounds utilizing a computer-based 
tearhire system, ?et r 1 rental c: extensive feedback were 

v ere c : m p a r e d . 



''■-frame cropra** or. time-credit leans with a 
. 1 vouch both KAR ar»;i non-KHR groups shewed 

feronees between groups did not prove to be 
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high $_s , the post-test differences among groups in both high schools 
were significant, with the regular pm^rar group learning the least . 

It may also be noted that the no-confirmation group generally showed 
a more positive attitude toward the program materials. 

The results in the literature, then, are conflicting and puzzling. 
A number of design and interpretation problems may lie behind some of 
the ambiguity. One suggestion of particular relevance stems from a 
study by MacFherson, Dees, and Orindley (1948-1949). In presenting 
the results of the study of skill learning they suggest an interpreta- 
tion which has implications both for those theorizing on the effects 
of feedback, and those reviewing or designing research in the area. 

The authors propose that the importance and function of KOR varies with 
the stage of learning. In the initial stages of learning, they contend, 
the directive or informational effect of Kor i^ important; after per- 
formance has stabilized the "incentive" function ol KOR assumes greater 
importance; finally when proficiency ha s reached a high. le vr el, overt, 
formal KOR seems to be of little valu*. This does nut hear, to he 
incompatible with, the view presented oirlie . \ hv h r 1 ' tree h and Ive IV. e r 
and others; that the effect of KOR varies with its redundance. 

A second suggestion relates item difficulty and the reinforcing 
effect of confirmation, 

Holland (19^5) points cut aft; r mv iov : :\c tue confirm:; 

programs with lev error r..tws . ; r used. ‘ n tht ether hand, 
utilized high error r . : "at or la !> multi to show an advantage 

f i rma t i c n g r ou r> . h tu e n t - ’ e c n f i do:: : c : t : : e 1 r own a n s we r s w : 
great in such, p re grams, "u.e rel.it : n ‘-c f.vm item difficult;, 
relative effectivyr.es- of c : r f i r m u L ; n ma y i f . indicative of th 
the reir.forcer in the urogram The n- r . e r rv. l » m urr-'u 
the •? to he correct either !, c:-v:r. e 1 ; e 1- told it i - ccrrm‘ c: 
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available answers, rhis presents only somewhat of a problem when com- 
paring no-feedback with feedback conditions. However, when schedule of 
rein f orcement is the independent variable, one night well question whether, 
for example, ’ 1 5 0 % answers available 11 is equivalent to "50% of answers 
observed. 1 ' In short: is the schedule of reinf orcenent to be defined 
formally: in terms of the number of answers available to the S_, or 
functionally: in terms of the actual number of answers observed? 

A second problem concerns the usage of the term "reinforcement. 11 
Traditionally, a schedule of reinforcement refers to the contingency 
involving a single operant class (roughly: "one kind of response"). 

However, the studies to be cited here involve "reinforcement" of classes 
of different, responses. Thus, a 50% partial schedule does not mean 
that one half of the emissions of responses in a certain class are 
reinforced. Rather, it means that half of the answers were present and 
available in the program, In point of fact, if the answers are indeed 
rein forcers , each one usually is reinforcing a different response. 

By way of preview it might be pointed out that the literature on 
schedules of reinforcement in pregrars resembles in one respect that 
reviewed in the previous topic: the findings of different investigators 
are in disagreement . 

Studies Shewing Schedule Effects 

In 1965, I.ubiin (1965) studied a lorge group of college students 
in a programmed osychcicgy course. Students were on schedules of rein- 
forcement including no confirmation, 100% confirmation, fixed ratio 50%, 
variable ratio dh F /. . She found that the YR 50% and the No-Re inf orcement 
group scored higher or, the criterion test than did those students under 
the continuous reinforcer, e treatment. The author suggests that "omission 

cf the answers ray have caused the subjects to lock for conf i mat ion of 
their responses In c ucceedir.g frames [p. 299]/' Presumably, this is tied 
somehc*. to better attending and, consequently, improved learning. Con- 
versely, the continue.:?; reinforcement group may net have engaged in these 
beneficial searching lehaviors and ~ay have.- learned little free, the frames. 
In addition, huhlir. suggests that the cost-test more closely resembles the 
r, - -answer program so that the regularly reinforced students may have been 
hand i capped when faced with it ~fter a continuous reinforcement format in 
the rregram itself. 

Hr umbo It 7 ar. d hicslcr ( I 5 ; pro: ; - e that in a lew error rate program 
res: onse requests without arswers prrvide the occasion for self reinforce- 
ment. Vsir.g a l"'-frare prrgram :r, educational test interpretation, they 
found that the l?ui, 2 If, and reinforcement groups did shew a differ- 

ence in behavior. The partial re in f or temen t groups clearly made more 
errors in the program and :r. the immediate post-test. lost-test scores 
■d e creased s 1 ^ t: e nu~:tr of c :m f i r-a 1 1 :ns and nu~ber of response requests 
decreased, a finding c : n t r a d i c 1 1 : g I u : I i n ' s . Tn e- 111% response request 
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and confirmation condition had L : e least frarrie errors } data similar to 
that obtained in the oilier study by Krumboltz and Weisrcan (1962). 

Time to complete trie program decreased as response requests and confirma- 
tions were, removed. The authors suggest that the findings o r the study 
support their hypothesis, though perhaps the measures of "self-rcinfor :e- 
rr.ent 11 are too few to be persuasive. 

In a study by Moss and Neidt ( 1969’) the problem was looked at in 
the context of information theory: both KOR and amount of certainty were 
varied. University and high school students sewed as Ss and a short 
(42-frame) adjunct s ty ie programi in insecticides was used. Decrements 
in learning were found both when items of information were omitted (i.e., 
lower percentage of KIR) and when uncertainty was reduced. The authors 
conclude that the effectiveness of KOR is intimately related to the 
degree of uncertainty: KOR is useful and important when uncertainty is 
hiph . 



Studies Sh owir l ■ Schedule E ff ect 



Glaser and Taber (1961) investigated the effects of partial Rein- 
forcement 11 using a synodic logic program for high school students. None 
of the four experimental groups (100% reinforcement, 50% fixed xatio, 50% 
variable ratio, and 25% variable ratio) differed significantly from each 
other on the criterion test, The authors suggest that the reinforcing 
ef fectivencss of feedback r.ay depend upon the age of the student, specific 
subject matter, IR and probability of correct response, 

Scharf (1961) used a symbolic logic program, with high school juniors 
and 50% and 25% variable schedules am. cl 100% and fixed schedules of 

confirmation. No significant relation : h L p w a ~ *cv;rd between schedules 
and post- test errors, program errors, cr prst-o-'St tim.e tc completion. 

The only important difference among groups seer-.d to be that the 50% 
variable schedule grout took the longest tim.e to complete the program. 

Krurrbolt- ar.d Veisr.an (I Q 6Z! also found no differ:?, 
test scores for variable vs. fixed ratio schedule? rrr for 
ar.d 110% ?cherRRs. There V3« n-< inter:*::;:? effect cith 
students made fewer frame errors prevra-s with more c; 

Dr is bill (196-) used Air Force men on a 52- frame p r : g r a :n r 
and r-cts of num'rers. He ; : ’i;R2 -VT, cr 191% feedback. N: sig 

difference w«s fcur.I ~etvecr. tht gain store? ter the two er' 

Fosenstock, l t no. (Irb5) v^cd four different schedule? of 
'w 1 1 r, ~ . “at mem. a* * : c rr^gra”! IRC/ , "h*/, * txe m, .b.*. vow *a: *e, nnu O 
: m r t 1 1 1 f.r, :v of ’"‘Suit? hid n. Z «-'t c m : * cr, '--uit t lea-r.ir.g, b 

fever pregra- errors occurred under conuiti'r. - if in o rented v ; * 

Using 6th- grade stuoir.t^, b 1 arh- a: d ;'y?u R * * R invent looted 
schedules :f r*. inf or e r, : with u c- r . : r t :r:er.w : r. :* t rr.gllsn money 
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system. The program was chosen for its difficulty, having a predicted 
high error rate. The study, therefore, attempted to answer criticism 
of previous studies to the effect that the failure to find differences 
due to various schedules of reinforcement was a result of using low error 
programs. (As noted earlier, again and again suggestions are made in 
the literature that "confirmation at the frame level may have little 
effect on student motivation except in situations when the student has 
real doubts about his answer as in unusually difficult f raraes , [Tabe r , 
Glaser, & Schaefer, 1965, p. 92].") In this study no significant 
differences were found among post-test, gain, or error scores for the 
various groups (100%, 67% VR, 33% VK, 0%, and "logical confirmation", 
this latter group having answers supplied on the terminal frame for 
each concept). The only significant difference was found in a sub-group 
taught by teaching machine. For that group, the difference in gain 
scores between the 100% and the 67% treatments was statistically 
significant. 

The authors, in their introduction, state "By and large the con- 
clusion suggested by (studies on confirmation) is that either (a) it 
is incorrect to regard con f inr.at ion in programmed instruction as 
equivalent to reinforcement as used in the sub-human context, or (b) 
these studies have fa. led to meet the necessary conditions that there be 
little chance for inter- frame cueing. If the latter conclusion is true, 
then varying schedules of confirmation, in the form of knowledge of 
results should manifest their differential effects where the probability 
cf error response is high. 

"A further possibility. . .is that the conceptual material found in 
most progranm.es dees net lend itself to random partial reinforcement 
[p. 13). M 

On the basis of their analysis of the literature and the findings 
of their own studies, the authors severely question the traditional 
observation that confirmation is a rein p orcer. The results of a later, 
more elaborate, study (Pysh, et al., 1969) again question the role of 
confirmation as a reinfcrcer. In that study the authors conclude: "In 

summary, it would appear that the pivotal assumption that programmed 
instruction's effectiveness derives from the explicit provision of KR 
in the form of a confirmation frame with which the learner compares his 
antedating response, requires a reappraisal [p. 62]. " 

2e lay of Confirmation 

Another probe to discover the reinfcrciig effects cf any consequence 
involves delaying the proposed reir.frrcer after a correct response is 
emitted. A large literature exists which, though r.ct entirely consistent, 
strongly suggests that delaying the presentation of the reinforcing con- 
sequence reduces its effect or. the behavicr upon which it has been mate 
c: n t i n r e n t . This literature has, f:r the erst part, involved infra-human 
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^Ss (although a fairly large sub-set concerns the behavior of retarded 
children). Evidence for delay noticeably affecting the performance 
of humans is less solid than evidence of such effects with lower 
organisms. (See, for example, Renner, 1964) 

Much of the research that seems to oe at all relevant to the 
issues raised in this paper involves feedback in test situations or 
in situations involving memorization of discrete associations. These 
results are not consistent nor always clear. Sometimes immediate 
feedback seems to be more effective than dees delayed feedback (see, 
for example, Sasssnrath, et al, , 1968); sometimes delayed feedback 
seems more effective, especially when long-term retention is being 
measured (see, for example, Sassenrath & Yonge, 1969; Sturges, 1969; 

B rackbi 11, 1964) , 

Delay produces effects . A few studies are directed specifically 
at the programmed instruction situalh n, Evans, Glaser, and Homme (1962) 
investigated delay using a program in symbolic logic* Delays in confirma- 
tion (ranging from 30 sec. to 5 min,) seemed to have only a little effect 
on criterion performance. The authors suggest that when the correct 
response is highly probable, the effect of confirmation may be minimal, 
a theme we noted occurring elsewhere in the confirmation literature, 

A study by Meyer (I960) involved presenting a 19- lesson program on 
Latin prefixes in English to 8th-grade students. One group had immediate 
feedback (answers were available in the text) arid another group had de- 
layed feedback (no answers were available in the text). The answers for 
both groups were "corrected" and returned to the student by the experimenter 
the next Gay. Students in the delayed feedback group committed more errors 
in the program, , The author suggests a lack of potential for self-correction 
in the form of the program used by tlv s group. Repetition of previous 
errors by these students, therefore, probably accounted for their poorer 
performance. Comparison of post-test scores for these two groups revealed 
a difference in the predicted direction just snort of significance (p = .06). 
The immediate feedback group shoved higher scores. 

The author concludes: "It was hynetbesized that immediate confirmatitn 
of correct res; crises and di s r tr.f irn.at . . r c f incorrect res rentes vculd lead 
to superior acquisition. Tr.e by or the si* was supported 'ey the data )r. 69]." 

Tie lay produces no e f f e c t . *’ : e r s r a (1966; u- e d a modi f i c a t : : r. c f the 
symbolic logic program used by Evans, with c: liege students. Lc f.ur, d a 
significant interaction effect of cclav :i feeTac! (i.t., time :r,m 
response to feedback exposure' 1 , and : . 1- f eed : a; k delay (I.t., time between 
end of feedback exposure one rre c entmt ion of next mnr.e ' on program errors 
but not cn criterion scores. 7h*. re were mo-e pi.gram errors in the feed - 
back - d e I a y , no post - ftedbacx delay c ; u ; r ; r. u: J in the no f t ed : a : k c e 1 av , 
post-feedback delay cor. ditirn. wcvtr, core i- r o rt an t ly , del ay of feedback 
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Other Kinds of Evidence 

A general observation can be n.-ide at this point: the evidence is 
weak that confirmation is a reinforcer, The results reviewed are at 
least inconsistent; if there is a trend it is toward showing no real 
reinforcing effect of feedback, 

Some lr secondary fl generalizations do emerge from the literature, 

For example* students with less, or no, opportunity to view 7 answers 
commit more errors in the. program. This finding seems explainable 
in terms of these in confirmation groups being able to peek at answers 
and to correct their errors. When the opportunity to peek is controlled 
for, usually there is an increase in errors noted in the confirmation 
group, That error or peeking has any effect on learning 'in terms of 
criterion test scores) is not at all clear. 

It could be argued that the reinforcing effect of confirmation is 
subtle and chat the experiments cited have not been sensitive enough 
to pick up an existing effect in terms of design or the measures of 
the dependent variables. Some suggestions have already been made about 
confounding or masking variables that ought to be considered, (For 
example, the argument has been repeatedly raised that the probability 
of correct response interacts 'with the effect of confirmation, an 
hypothesis to be returned to later,) 

Organism Variables 

Permanent or momentary organism-centered variables (e,g,, anxiety, 
I,Q,, sex, age, achievement motive) might confound experiments in which 
ether variables are being manipulated. Assuming that the literature 
search made in the preparation of this paper was fairly comprehensive, 
one would conclude that the only extensive research on such variables 
has been conducted by Campeau whose w'erk vas reviewed earlier, A major 
variable in her research has been anxiety, specifically test anxiety 
as measured by the Test Anxiety Scale for Children, The author contends 
that accentuating the test-like features of the situation by omitting 
answers in the program should adversely affect high anxiety S_ s , Further- 
more, a comparison of high and lev anxiety Sis under feedback conditions 
should reveal higher achievement scores for the high anxiety group 
f imp lying, in the context of this paper, that feedback may be reinforcing 
or at least more reinforcing, to anxious students). The difference found 
in Campeau’ s studies were r. t significant for the m.ale S. s . High anxiety 
girls did somewhat better, but not significantly tetter, on immediate 
post-tests than did lev anxiety girls when bet h groups had received feed- 
back in the program. High anxiety girls in the feedback group showed 
dramatically better gain scores cnan these in the no-feedback group. 

Toe evidence is net very clear, however, that feedback is mere reinfcrcin 
to high than to lev anxiety students, 
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Though a number of writers (e.g. , see Taber, et al. , 19o5, p. 170) 
suggest that factors such as age, motivation, and I.Q. may well interact 
with feedback, little research has been directly aimed at investigating 
such relationships. (Though there is a small literature on student 
characteristics and programmed instruction, see, for example, Woodruff, 
et al. , 1966. ) 

Task Variables 



A second set of variables which night effect the status of feedback 
involves the task itself: the kind of task (e.g,, notor skill learning 
as contrasted with verbal discrimination), the degree of interaction of 
the parts of the task (e,g«, learning rote material in which the components 
have little interaction with each other vs., learning concepts which are 
related) or task complexity. For example, some pilot work by the authors 
(Geis & Chapman, 1970) utilizing a program teaching Russian script (i.e., 
the cyrillic alphabet) suggests that students are more likely to request 
feedback when the task requires production, in this case writing a letter, 
than when a discriminni ion alone is called fnr, as when the student is 
asked to choose the better of two letters. More generally stated: feedback 
might be more reinforcing when one is executing a complex motor coordination 
than when he is merely recognizing a correct item in a choice situation. 

This may be related to the idea that the probability of error interacts 
with the reinforcing effect of feedback, since the chance of making some 
error is usually increased when a chain of responses is called for, as in 
the case of production. 

Kinds of Feedback 



A third area might be called "k’nds of feedback." Certain types of 
feedback may, in an absolute sense, be mere reinforcing than others, or 
may be more reinforcing under certain conditions. For example, given a 
two-choice discrimination task in a program trove, the student might be 
reinforced by an indication of '’correct response”, as a light going on 
after he emits the right answer. Failure to produce the "correct” signal 
is logically equivalent to producing an "incorrect" signal, in the twe- 
thoice situation. (There is some indication that feedback for right and 
no feedback for wrong answers is not complete y equivalent to identifying 
the answer as right - wring. Kiwever, generally speaking the feedback 
for correct responding in a two-:h:ice equation see~s t: exhaust all the 
possibilities. .» 



At the other extreme mizht te a c 
to pren ounce a French v*rd rr to write 

hand, a mere elaborate :ee oracx sy £ te~ 
dimensions of the student’s res: on-e v 
providing a correct comparison model f 
reinfercir.E. 
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"Kind of feedback" might also refer to varying contingencies of 
the same feedback, i,e,, the same information. (Thus, in a study by 
Anderson to be discussed later [Anderson, et al, , 1970], one group 
received answers only after they had made an incorrect response while 
other students received feedback only after they had made correct 
responses • ) 

Branching programs often provide elaborate and varied response con- 
sequences, In examining possible evidence for the reinforcing effects 
of various kinds of KOR it might be appropriate to include the branching 
literature. Unfortunately, there seems to be no research specifically 
directed to explorations of kinds of branches, number of alternatives 
in a branching system, etc,, (This seems a fallow’ area for research.) 

The assumption is conmorly made that a branching program, especially a 
computer-assisted one, is bound to be superior to a more pedestrain 
linear program, In summarizing his review of the area, Holland (1965) 
noted no significant advantages had been demonstrated for branching 
programs, Tvo years later Anderson (1967) came to the same conclusion 
despite additional research that had been conducted in the intervening 
time. There may be advantages to branching; certain kinds of branching 
consequences may be reinforcing. But at present there is little evidence 
to that effect. 



The research literature on kinds of feedback in programmed instruction 
is small but interesting. 

Kind of f eedback affects learning . Krumboltz. and Bcnaw-itz (1962) 
varied the form of confirmation by using their educational test con- 
struction program. The "isolation" approach involved presenting as 
feedback the word or phrase that was desired as a response. The "context" 
approach presented the confirming response by repeating the relevant part 
of the stimulus frame, The experimenters found no differences on a sub- 
test of knowledge of technical terms, but a significant advantage was 
found fot "context" confirmation on a sub-test of applications of test 
construction knowledge. The authors caution that the findings are for 
a small group of Ss, Nevertheless, there is a suggestion that kind of 
feedback may interact with the development of certain kinds of terminal 
behaviors and not others, 



Gilman (1968) has investigated tie effect of various kinds of feed- 
back in a computer-assisted instruction system. He points cut that "if 
there were no purpose tc- feedback, other than to prrvlde the student with 
reinforcement, statements such as 'you are correct' sbculd prove equally 
effective as confirmation of a correct answer [p. 2]." University upper- 
classmen were taught 30 general science concepts by means of a CAT adjunct 
self-instructional system, using a multiple choice forma:. Various modes 
of feedback were used: r.o feedback; "correct" or "wrong 11 ; feedback of 
correct response; feedback appropriate to the student's res: :rse; er.c a 
crmbir.aticr. c r the three latter modes, Students repeated items -which were 
-isse: until a perfect run thrrueh was retained, 7r,e n:-feedback group 



o 

ERIC 



1 



Geis & Chapman 10 

and the "correct' 1 or "wrong" group performed less well on the program, 
making a significant/ greater number of responses and requiring a 
greater number of iterations of the program in order to reach criterion. 
On the post-test, the combined feedback group scored significantly higher 
than did the others. 



These two studies seem to suggest that more elaborate feedback may 
be more reinforcing, cr at least mo r e et receive in some way , in changing 
student behavior, However, other studies do net support this Hypothesis, 

K ind of feedback do e s r.ot affect lea rning. McDonald and Allen (1962) 
varied the kind of confirmation in a program which taught a game similar 
to chess or checkers. The variations in confirmation included: no response 
request and no confirmation; response request with correct response as 
confirmation; response request, correct response and an explanation of 
correctness or incorrectness as confirmation. The experimenters found no 
differences among groups on immediate and delayed criterion tests, 

Bivens (1954) used a short program, in elementary set theory wit a 89 
8 th-gr ade students. Confirmation wa=. offered in simp le and comp lex forms. 
The. simple form was a presentation cf the explicit desired response, Tne 
complex form was an example of a different but similar problem already 
solved, All students had the complex form of confirmation available. 

Those in the "simple" form condition were told to check the answers against 
the simple answers and merely rerd the Ovher selves problems, Tncse in the 
"complex" group were instructed to search through the solved problem to 
determine if their own answers were correct. There were no differences 
found between the group? on the criterion test after learning. 



Anderson, et al. (1970) use-: 
a computer and a program or. Jiagr 
experiment, using several groups, 
only after a correct response ha: 
response had been emitted, or ( J ; 
correct response but the _S "loop* 
responses. Criterion test scores 
the looped groups. The only £ro\ 
was the nc- feedback group. Cf i: 
cant difference was found be twee: 
the latter students underwent mu; 
Furthermore, there was ro evi..er, 
as corrective feedback. The gr:: 
ma ce d i c n o : per f : m sign! f i : an t 
c t h t r K T R groups, Ag r. i n , w: . a r e v • 
was ro t e c t ab 1 i ? h e c . Th e YS. h- : n 
S a *". e * e vc ~ as t r, e ctr.cr groups. 
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In a second experiment by the seme authors, k he 0 and 100?. groups 
vere again used. Various groups of Ss who always received KOR were 
exposed to one of the following additional variations: was forced to 

repeat wrong frames until he made the correct response; after each of 
four sections (about 25 frames) in th? program _S repeated any wrong frames 
in the preceding section until he made a correct response; when he made an 
error $. saw the frame and KOR for 15 sec.; .S was presented with frame and 
answer but was instructed to respond before looking at KOR. (This last 
group was similar to groups of students using programs outside of experi- 
ments and without mechanical presentation devices in that "peeking’ 1 was 
possible.^ Again, the 100^ group proved to have undergone the most 
effective procedure both in terms of reduced number of frame errors and 
increased criterion scores. And again, r.c difference in favor of the 
more elaborate feedback procedures was found in this study. 

Melaragno (I960) investigated nega l ive feedback using a set of 50 
multiple choice items for teaching the names, uses, and meanings of five 
logic symbols. The material resembled programs of the cay, having easy 
small steps. A small group of junior college students acted as Ss t 
Five ambiguous items with no correct answers were inserted to determine 
the effect of negative reinforcement on post-test score. Members of all 
groups were shown a green light after each response to an item In the 
program except for the five ambiguous ones. The green light indicated 
that the previous response had been correct. (S. received the green 
light, "correct/ 1 signal even if he had made an error on the fram.e.) 
Following responses to the ambiguous items: Group I saw the green light 
after each response to the ambiguous items, which indicated they were 
correct. For this group the ambiguous items were spaced throughout the 
sequence. Group II saw a red light indicating responses to the ambiguous 
items were wrong ( Again, the five items vere interspersed throughout 
the programs Group III also saw "incor re c t" feedback lights cn the five 
items which were in this case massed at the middle of the total set. 

All groups were given a 45-questicn criterion test upon completion 
of the learning sequence. G r o u p III had lover pest- test scores than the 
other two groups. The author concluded that seme spaced negative rein- 
forcement does net impair learning, but the massed negative reinforcement 
seems to. 



Prompting and Cueing 

In pursuit cf the specific conditions under which confirmation might 
re reinforcing the literature or. prompting was examined. Toe conjecture 
was one already repeatedly discussed in this paper: namely that the rein- 
forcing function of KCR ray depend in good part upon the probability that 
emits a correct answer. Frames with a high degree of prompting reduce 
the chances that an erroneous response will occur end, therefore, confirma- 
tion following such frames should be minimally reinforcing. 
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There is a large literature on prompting versus confirmation, much 
of it involving nor-programmed materials (e.g., Cook & Spitzer, 1960). 

A comprehensive review by Aiken and Tau (1967) examined three types of 
learning: verbal learning, perceptual learning, and sigrvi monitoring. 

The authors conclude response prompting is as effective as, or is more 
effective than, response confirmation. (A possible exception, the 
authors note, is discrimination 7 earning.) While cautious in their 
conclusions and extrapela ticns , trie authors stress that one should 
neither ignore the potential importance of antecedent stimuli in con- 
trolling behavior and effecting learning, nor subsume all consequent, 
response contingent, events under the rubric v rein forcer , ” 

From the literature, two opposing suggestions concerning prompting 
in programs emerge. The first is derived from paired associate learning 
studies in which the superiority of prompting cvei confirmation seems to 
have been established. Extrapolating to programmed materials from such 
data, one would recommend heavily cued frair.es> The second view (e.g,, 
Margulies, 1968) suggests that over-prom.pt ing in a program, especially 
one involving non-rote materials, nay reduce the effectiveness cf 
learning since it eliminates searching, problem solving and some of 
the other (probably covert and assumed) behaviors that the programmer 
intended his student to engage in. 

There are about half a dozen studies on the issue of prompting 
which do utilize programmed materials. 

No Effect 

Silberr.an, et al, (1961) used 61 teaching items on topics in logic 
with junior college students. A version of the program, providing con- 
firmation after each item was contrasted with me which presetted the 
same items cf information one at a time in statement form, and with one 
which presented the same information in paragraph form allowing review. 
Trough the two "prompted 1 * versions took less time to complete than the 
confirmation version, there were no differences among groups on post- 
test scores. 
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decoding rule vita each problem. confirmation involved S_ seeing the 
decoded sentence after attempting to decode the problem sentence. Scores 
on post-tests, retention tests and transfer tests were highest when 
prompts were given, about the same when piompts and KOR ver: given, 
lower when KOR alone was given and lowest when neither KOR -.or prompts 
were p vided. The authors conclude that KOR is effective but only 
when it is not redundant (:,e, , KOR plus prompts was no more effective 
than prompts alone), 

Anderson, et al, (1968) modified the Hell--: i-Skinrier psychology 
program for use with 106 high school teachers in an educational 
psych logy course, The first ‘,052 frames were used, with one addi- 
tional ;rompt added to about 9 % of the frames of the ’’prompted" version, 
Time to complete the program and post-test scores were compared for the 
prompted and a r. : n-prcm.pt ed version. S_s in all groups could look at the 
confirmation on tne page containing the next frame. Ss in thn prompted 
version took significantly cess time and prefaced lower test scores. 

The authors suggest that there is a limit to the efficacy of prompting, 
that lim.it having been reached with this kind of program. 

The authors related theii findings to two et c r bodies of research 
on prompting; Cook’s work referred t : earlier and Holland’s research 
cn "blackout ratios” (Holland, 1965), They propose that their findings 
complement Holland's contention that the response in a program must be 
contingent upon attention to critical material the frame, Anderson 
states: "learning is reduced when the prompts r*. e of suen a nature that 
it is possible for th» student to respond correctly without attending to 
the cues fp. 93]. ” 

Although a study by LeFurgy and fissercr. (196:) dees net utilize 
pr: gramm.ee material it seems highly relevant here. Tne authors utilized 
cmcept care with ?e one trio figures t: teach 31 young girls the concept 
"the cr.e with the meat sides is the correct cr.e.’’ ?r.e group received 
confirmation of correct choices ''actually, pressing a button) cn acquisi- 
ti;r trials, Another grove saw a 1 icht go cn ir front of the correct 
figure on the card (”: rcm.pt int”; tier, raue a choice. A third group could 
decide cn which procedure, pr-mptinz :r urn f i mat ion , they would like to 
follow :n each trial, No difference? (trials t: oriOerirn, error*, time, 
etc.) were four.: among v.e groups during accyuisiticn, However, tie free 
choice group vas superior cn the generalization post-test and in their 
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importance of feedback clearly depends upon the adequacy of pricr 
prompting. 'Vith a well-prompted program of the Skinner type, students 
may frequently pay little attention to the confirmation/correction panel 
since the program is sufficiently veil cued so that they are generally 
certain of the correctness of the response they have made [ Lumsdaine , 

1961, p, 490], 11 He further suggests, specifically, that "at any point 
during acquisition, the strength of unconditional cues or prompts 
provided to the ^earner should satisfy the dual conditions of being: 

(1) just sufficient to elicit the correct response, but (2) no stronger 
than is required for this purpose [lumsdaine, 1961, p, 4821." 

The point which emerged early in the paper seems to be strengthened 
and clarified with each additional section: if KCR can function as a re- 
inforcer at all it is likely to cc so under special conditions which 
involve low probability of correct response (or, conversely, high probability 
of error) f Lumsdaine seems to be suggesting that the programmer produce 
materials which are delicately designed to "keep the student on the edge", 
barely cueing the correct answer, 

Student l‘se of Feedback 



One of the most obvious ways to discover the actual reinforcing 
effect of KOR would t: to make it available tc the learner and notice 
vnether cr not he takes advantage of its availability. If a re inforcer 
is defined as strengthening the response which precedes it, and if KOR 
is indeed a reinforcer, then the response of producing r’.CR should be at 
high strength. Or as Anderson h 1 9 6 7 ) puts it: "If an explicit ’observing 
response* were required cf students ir. order for them tc receive KCR, then 
the rate at which this observing response occurred would be by ce : ini ticn 
the measure of strength cf KCR as a reir.foicer ;p, 1-9). 11 



Melching (1966) used a 56 4- frame linear nr reran :r< magnetism wit:. 

17 enlisted Arm;, personnel, All answers had neon deleted fr:m tr.e 
program which was then cdminlstered individually t: each student. 
perimerter, sitting opposite , provided feedback 'the printed answer" 
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(1) On the average students checked far less than 100/s cf the answers, 

(2} Students varied from one to another on the percent of answers they 
checked * (unreported data indlcatt that each S_ is consistent in his 
checking rate over a variety of programs), (3) Clear, significant and 
positive correlations were ^tt&Lned fc: tween erroneous responding and 
checking. Thus, though the 'becking rates differed widely from student 
to student, with only a few exceptions, the probability of checking, 
regardless of base rate, is higher after a student has emitted an 
erroneous response than when, he has been correct. 

- . Anderson's comment on Melching’s data holds for these findings as 
veil: r r . . the cnly decent thing to do in the face of these data is to 

question the assumption that KCR is a reinforcer [Anderson, *967, 
p, 149], " Nevertheless, that bread statement seems to require an 
amendment. ’Jnder certain circumstances, namely after an error has 
been committed, the probability of observing behavior is raised and, 
by Anderson's definition, the printed answer may be said to be reintoremg, 
(Little attempt has been made in this paper to answer the question: Vhat 
is being reinforced by feedback? While the ill-defined term learning 
intuitively seems like the proper and relevant answer, "the response just 
previous to feedback" is less assailable, It is assumed that the 
observing response is sor.ehcv related to learning and cc continued 
participation in the program, but the relationship is not clear, 

Summary and Conclusions 

Knowledge of results is most frequently cited as the reinforcer in 
self-instructional systems. The printed answer in a programmed text, for 
example, is supposed to reinforce the response the student emits previous 
tc observing that answer. Seme other possible reinforcers were briefly 
discussed in this paper before the literature cn KOR in self-instruction 
wa? selectively reviewed* The review was organized as a searen for 
evidence that KCR might appropriately be callec a reinforcer. Studies 
: :mpa r ir.z programs with and without feedback were examined; the weignt 
of evidence from these global studies was that feedback did r.- enhance 
learning, as measured by immediate rrst-test scores or by retention 
tests. In at least ore case there seem-rd to 1 e a decrement ir. p erfor~.ar.ee 
traceable to the presentation of feedback. One recent and sophisticated 
stud v ■* i . e . , Am. c e r s o r. , s t a 1 * f 1 9 ■ C ) dec ? r. c z * e a r a z v an t a g £ s for reec — 
ra:k and thereby implied that KIR may be a rein forcer . 

o tudieE in which "schedules of reinforcement' were vsriec similarly 
failed t: show effects that -would be expected if KOR were acting as a 
reir.frroer. / K:we**er, high error scores within the pregram are con- 
sistently noted vher. percent KOR is reduced.) 

Cne - a ; : r study involving :elay :: KOR ■'!«€., Meyer, 19fl) did re- 
::rt tre effect expected when delivery of a reir,f:r:er is de*a ec. 

;tr,er studies or, delay d : not replioate inis finding. 

hirer grained analyses :: stolen! ••.avior and KOR regir. to revea^ 
s;e:*fi: renditions under which =-e€-= or : e acting a? a reinforcer. 
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A few studies scattered throughout the literature report on manipulation 
of subject and task variables and cf kinds of feedback,. The results of 
these and the remits of studies in prompting vs_ confirmation and student- 
controlled feedback suggest, each in a different way, that KOK may well 
be a reinforcer when uncertainty, or confidence, or probability of 
emitting an incorrect response is high. 

It is clear that the printed answer (or its analog in other media) 
is not globally and automatically a reinforcer. The review provides 
the springboard from which jne might jump into broader questions such 
as how’, when, and why information on one's own performance in a learning 
situation becomes reinforcing. 



Footnotes 

‘Preparation c-f this paper ’was supported in part pursuant to 
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F.L. 85-86* , as amended. This paper has teen accepted for purlica- 
t ; : i n t h e Educational Technology Journal , 

“Dr. Geis is new at the Centre fr Learning and Development, 
McGill University, Mr. Chapman is at the Center for Frcgram-ed 
Learning for Business, The University cf Michigan. 




Ga is 6 Chapman 



25 



Re ferer.ces 



Aiken, E . G, , 6 Lau, A, V, Response prompting and response confirmation; 
A review of recent literature. Psych olog ical Bulletin, 196 7, 68.(5), 
330-341. 

Alter, M. , i Silverman , R, Tne response in programed instruction. The 
Journal of Programed Instruction . 1962, 1,(1), 55-78, 

Anderson, R, C, Educational psychology, Annual Review of Psychology , 
1967, 18, 129-164, 

Anoerson, R, C, , Faust, G. W, , & Roderick, M, C. Overpronp ting in 

programmed instruction, Journal ~>f Educational Psychology , 1968, 
59(2), 88-93, 



Anderson, R, C, , Kulhavy , R. V., 8 Andre, T, Feedback procedures in 

p r o g r acime d instruction, Jour .al of Educational Psychology , ir. press. 

Annett, J. Feedback and Kumar. Behavior . Baltimore: Penquir. Books, 1969. 

Becker, J. L, The effect of withholding reinforcement in autoins true tional 
programs. In G. D, Ofeish 4 V, C. Meierhenry (Eds.), Trends in 
Programmed Instruction . Washington, D, C • : DAY I , National Education 
Association and National Society for Programmed Instruction, 1964, 

Serlyne, D, E, Conflict, Arou sal , an.-! Curiosity . Sew York: McGra r w-Hil 1, 19 60, 

Berman, L, Tne experimental analysis of per f orrr.ar.ee on programmed 
materials. NS PI Journal , 1967, 6/10), 1C-13. 

Bivens , L, V , Feedback complexity and self-direction in programmed 
instruction. P sychological Reports , 1964 , i£ t 155 -160, 

Blank, 5. S., i Fvsh, F. The effects of varying confirmation schedules 
in e high error rate program. Canadian Psychologist , 196', £ a { 1 ) , 

12 - 16 . 



ma, F. J, Effects of delay of inforr.ation feedback and length of 
post feedback interval cn linear programmed learning. Journal of 
Educational Psychology , 1966 , 3 7 ( 3 ? , 140-145 • 



B-rackhill, Y, , Vagner, J. I . , h V i 1 ? : n , D. Feedback delay ar.d th? teach- 
ing machine. Fsycn elegy in the Schools , 1964, _1(2), 148-156, 

Campeau, P. 1, Test anxiety and feedback in programmed instruction, 

J : u : n a 1 tf Experimental Psy:.v:I-gy , 196-8, 59 (3), 159-163. 



2::k, J. 0. , !i fpitzer, M. I . 

ccr.f irmat i:n in p&irei-ass 
PsycholcA" t 196 3 , _59='i • , 1 



elementary report: Prompting versus 
ate learning. Journal cf Experimental 
2 ~ 6 . 




* / 

J 



Geis & Chapman 



26 



Dri skill, V, K„ Partial and continuous feedback in a linear programmed 
instruction package. In G. D. 0 flesh & V. C. Meierhenry (Eds.), 
Trends in Programmed Instru c tion . Washington, D. C. : D/ VI , 

National Education Association and National Society fer Programmed 
Instruction, 1961, 

Evans, J. 1. , Glaser, R. , & Homme, L, E. An inves : igat : on cf "teaching 
machine" variables using learning programs in symbolic logic. 
Journal of Educationa l Resea r ch , 1962, 55(9) , 433-152. 

Feldhucen, J. , £ 3irt, A, A study of nine methods of presentation cf 
programmed learning material, Journal of Educational Research , 
1962, 55(9) , 461-466. 

Ferster, C . B . , & Perrott, M . C , Behavi o r Principles . Nev V c rk : 

Applet on-Cer. tury-Crof ts , 196 S, 

Gagne, R, M, The analysis of instructional objectives for the design cf 
instruction. In R, Glaser (Ed.), Teaching Machines and Programmed 
learning, U : Data anc Directions . Washing ten, D. C.: DAVI, 
National Education Association, 1965. 



Geis, G. 1., Chapman., R . , Smith , C . W , , & Moore, C. J, I r.p roving 

Instructiona l Systems . Mimeo. Foreign language Innovative Curricula 
Studies, Ann Arbor, Michigan: T.S.O.E, Grant J ; 3-7-7 0-143 1-4369, 1969. 



Geis, G. L, , & Chapman, R. Feedback in rrogramm.ed instruction. ?r 
Report. Center fer Research cm Language and language Eenamcr, 
0,E. Contract 0EG-0-9-C977- 1 -374 3(01**; , January, 1970, 



r e s 5 



Geis , C. L. , 



printed 

£-11. 



J a c ob s 
ansver 





c 




Geis, G, 1., £ Enapp, $. A note cn non fun ‘ : ion 
program.. Journal cf Programed Instruct!-:. 



Gilman, S. A, A ccr.pariscr cf several feedback methods for corrector , i 
errors by ccr.puter-assiste c instruction. Paper presented to the 
American F s v e he 1 c c : ■: 1 Association, Als : ERIC document: 

EM2-2S7, 






sviorsl science base for instruct 





uvaser, R, 



Pr eg r ante d 
Fro rrammed 




Investigations :f the Characteristics 





E d s . 
1 - f 2 



:k in aut cm 









^ _ f a 7 w 



Gels 6 Chapman 



27 



* 



Got kin, L, G, Trie machine and the child, AV Communication Review , 1966 , 

14(2), 221-241, 

Hershberger, W, Learning via programmed reading, Palo Alto, Calif,: 

American Institutes for Research, Contract Ho. NONR-3077 (00) , 

Office of Naval Research, Technical Report AIR-C28-7 ,/63-TR No, 5, 

July, 1963, 

Hershberger, W, , 4 Terry, D. F. Typographical cueing in conventional 

programed texts. Journal of Applied Psychology , 1965, 49(1) , 55-60. 

Holland, J. Research on programing variables. In R. Glaser (Ed,), 

leaching. Mac hines and Programmed Learning, II: Data and Directions . 
Washington, D, C.: DAVI, National Education Association, 1965. 

Hough, J. B. f 9 Revsin, B. Programmed instruction at the college level: 

A study of several factors influencing learning. Phi Delta Happan , 

1963, 44(6), 286-291, ^ 

Human Development Institute, (J. Berlin 6 J. B. Wvckoff) Genera l 

Relationship Improvement Program . Human Development Institute, 1966, 

Jacobs, P « I., & Kulkarni, S. A test of so ce assumptions underlying 

programmed instruction. Psychological Reports , 1966, 1J5, 103-110. 

Keller, F, S. Neglected rewards in the educational process. Ir. 

Proceedings of the American Conference of Academic Leans . L c s 
Angeles: 1967. Pp , 9-22, 

Krur.bcltz, J. D. , 4 3onavitz, B. The effect of receiving the confirming 
response in context ir. programmed material. Journal of Educational 
Research t 1962 , JR5(9) , 472-475. 

V r unb o 1 1 z , J . D . , 4 Ke 1 s 1 e r , C . A 
and programmed instruction. 

196 5', 3(2) t 9il4. ... 

Kr unbolt* , J. D. , 6 Veisman, R. G. The effect of intermittent confirma- 
tion in programmed instruction. Journal cf Educational Psychology, 

1962 , 53(6}, 250-253. 

LeFurgy, V. G. , 6 Sissercn, J* A. Effect of prompting, confirmation ar.d 

free choice upon children’s acquisition, verbalization and generalize- * 
tier, ci a concept. Psychcr.cmi c Science , 1968, 1_2(6) , 277-278. 

Lublin, S. C, Reinforcement, schedules, scholastic aptitude, autonomy 

need, and achievement in a programmed course. Journal cf Educational 
Psycho l ogy , 1965, 5666), 295-302. 

Lumsdaine, A. Seme conclusions concerning student response an: a science 

cf instruction. In A. Turns dal me Id.), Studen t Res; cr.se ir. Programmed 
Instruction . Washington. 1. 6.: Notional Academy cf Science ar.d 
National Research Council, tub Heat ion 1961. 

' / . 
<u J 




The partial reinforcement paradigm. 
J ournal of Programmed instruction , 



Gei s $ Cnapman 



28 



MacFhe rson , S. J., Dees, V,, & G-rindley, C. G, The effect of KR on 

learning and performance II. Some characteristics c-f very simple 
skills. Quarterly Journal of Experimental Psycholog /, 1948-49, 

JL, 68-78. ^ _ 

Margulies, S., & Speeth, K. What determines wnat the student learns? 
Programmed Learning and Educational Technology , 1968, 5_ 4 ) , 
255-270. 

McDonald, E, J., Allen, D, W. An investigation cf presentation, 
response and crrectior.al factors ir. programmed instruction. 
Journal of Educational Research , 196 2, 5 5 (9) , 502-507. 

Melaragno, R. J. Effect of negative reinforcement in an automated 

teaching setting. Psychological Reports , i960, _7, 3S1-384. Also 
in J. ?. DeCecco (Ed.), Educational Technology : Readings i n 
Prcgr armed Instruction . Nev York: Holt, Rinehart 4 Winston, 1964. 

Me lc nine . W , Vi, PI under a feedback schedule. NSPI Journal, 1966 , 

5(f), 14 - 15 . 

Meyer, S. R. A test of the principle of "activity 1 , "immediate 

reinforcement ” , and "guidance"; as instrument ed by Skinner's 
teaching machine. Dissertation Abstracts , I960, 20_, 4729-4730, 

Michael, D. M. , 6 Maccoby, N, Factors influencing the effects of 

students’ participation on verbal learning 2rcn films: Motivating 
vs, practice effects, "feedback", and overt vs. covert responding, 
I n A , hums c sine < he , } , Stuctr.t Response in Programmed Ins true tier, . 
Washington, 0. C.: National Academ.y cf Science and National 
Research Council, Publication r: 94 3, 1961, 




Me ore , 



Ameers 





V 



rss, D . : , , i 




rrer 






r ; e - 



o 

ERIC 

air.> l M l ilT7nnl71 . 1.1 



C-eis S Chapman 



29 



Pysh, F . j Blank, S. S., & Lambert, R. A. The effects of step size, 

response mode and KCR upon achievement in programmed instruction. 
Canadian Psychologist , 1965, 13(1), 49-64, 

Renner, K, E, Delay of reinforcement; A historical review, Psychological 
Bulletin , 1964, 61/5), 341-361. 

Ripple, R, E. Comparison of the effectiveness of a programed text with 
three other modes of presentation. Psychological Reports , 1963, 

12, 227-237. ' 

Rcsenstock, E. K. , Moore, J, W. , 4 Smith, V. I. Effects of several 
schedules of knowledge of results or. mathematics achievements. 
P sychological Pveports , 1965, 17 , 535-541, 

Sassenrath, J, K. , et al. Immediate and delayed feedback on examination 
and immediate and delayed retention. California Journal of Educa - 
tional Research , 1968, 19(3), 226-231. 

Sassenrath, J M. , & Yor.ge, G. D. Effects of delayed information feedback 
and feecoackt cues ir. learning or. delayed retention. Jojrr.al of 
Educational Psychology , 1969, 61(3) , 174-177. 

Scharf, E. S. A study of partial re info ream, ent on behavior in a 

programmed learning situation. In R. Glaser (Project Director) 
Investigations of the Characteristics of Programmed Learning 
Sequences . University of Pittsburgh, Programmed Learning Lab., 1961. 

Si Iceman, K. F. , Kelaragnc, ?. J . , S Coulsor. , J. E. Confirmation and 

prompting with connected discourse material, Psychological Reports , 
1961, 2, 2 35-2 38. ~ 

Skinner, B. F. The Technology of Teaching . Nev York: Apple t on -Century - 
Crofts, 1968. 

Smith, P. E, P., Bret hover, D . , i. Cabot, R. Increasing task behavior. ir. 
a language arts program, by providing reinforcement. Journal of 
Experimental Chile Psychology , 1969, 45-62. 

Sr i t h , 2 . E . F . ( F c . ) Successive language Discrimination Program . An r. 
Arbor: Ann Arbor Fvb Ushers, 1964 . 

Sturges, r. 7. Verbal retention ar a function cf the informativeness 

and delay cf informative feed bark. J ourna l of Bduatior.al Psychology , 
1 969, 6cVl) , 1 1-14 . 

Sullivan, K . J., raker, R. L . , 5 fchutz, R . E . Effects of intrinsic and 
extrinsic reinforcement contingencies or. learner performance. 

Journal of Experimental Psychology , 196 7 , 5 S f 3 ; , 165-169. 



Geis ■& Chapman 



Swets, J, A., Mailman, S. H, , Fletcher, W. F . , 6 Green t D, M, I. eurnit 
to identify nonverbal sounds; an application of a computer as a 
ter chi ng machine. Journal of the Acoustical S ociety o f America , 
1962 , 34(7), 128-935. 

Taber, J. I., Glaser, R. , & Schaefer, F. H. Learning and Programmed 
Ins t rue t icn . Reading, Mass.: Add ison-V.'es ley Publishing Cc . , Inc 
1965. 

Ugelow, A. Motivation arid the automation of training, a literature 
review. Final report. Behavioral Sciences Laboratory, 6570th 
Aerospace Medical Research Laboratory, Aerospace Medical Civisio: 
Air Force Systems Command, Vr ight-Pat terson Air Force Base, Ohio 
Technical Documentary Report 4M?J. -TCR-62-15 , March 1952. 

White, R. W, Motivation reconsidered: The concept of competence. 
Psychological Review , 1959, 56(5 ) , 297-533. 

Witt rock, M, C,, & Twelker, ?. . . I rotr.pt ing and feedback ir. the 

learning, retentior and transit.: of concepts, British J o urnal 
of Educational ?syc..:ln g , . 964, 54(1) , 10-18. 

Woodruff, A, 3., Faltz, C., 5 Wagner, I 1 . Effects of Learner 

characteristics on programed learning perfcrrar.ee. Psyc h ology 
i n the schools , 19 £ , 2*^), 72-"", 




